%%This is a very basic article template.
%%There is just one section and two subsections.
\documentclass[conference]{IEEEtran}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage[pdftex]{graphicx}
\usepackage{url}
\usepackage{booktabs}
\usepackage{flushend}

\begin{document}


\title{Classification of art paintings by genre}

\author{\IEEEauthorblockN{Mateja Čuljak, Bruno Mikuš, Karlo Jež
  and Stjepan Hadjić}
 \IEEEauthorblockA{Faculty of Electrical Engineering and Computing \\
 University of Zagreb \\
 Unska 3, 10000 Zagreb, Croatia \\
 Email: \{mateja.culjak, bruno.mikus, karlo.jez,
 stjepan.hadjic\}@fer.hr}
 }

\maketitle


\begin{abstract}
This paper offers an approach to automatic art genre classification of paintings.
Development of machine learning algorithms and increase of overall computing power 
improved speed
and efficiency of feature extraction from digital images and with it opened a whole 
new set of possibilities
in classification of visual data such as paintings and other visual art.
Automatic classification is useful
in large database processing (e.g. museums) and could be used as a commercial
application on mobile platforms.
Six genres are classified in the paper: realism, impressionism, cubism, fauvism,
pointilism and na\"{\i}ve art. Some of the genres have now been tested for the first
time. Used features are described as well as a measure of their usefulness. Rate of
success for different
classifiers is given. Accomplished results are similar to related work
results.
\end{abstract}

\IEEEpeerreviewmaketitle

\section{Introduction}
One of the tasks that people perform well is classification by type, e.g.
classification of art works in genres, art movements or periods. However,
it is not 
quite simple to describe the thought process of making a choice for a specific genre.
Because of that, the task of painting classification into genres is
traditionally entrusted to human experts.

Lately, along with development of machine learning algorithms and image
processing, there have been attempts to perform this task automatically.
Automatic computer classification is inspired by human perception of art and it is guided by
art critics principles of distinguishing works of art. It has been shown that
people perceive art genres through different elements of painting -- colors,
shapes, borders (e.g. motifs in painting, color palette, used techniques,
brush stroke style, color mixing, edge softness etc.). There is no unique
mechanism for visual art interpretation, therefore painting elements are interpreted
separately \cite{zeki1999inner}. Art genres often include a large number
of painters, so a genre is distinguished from others by properties similar to
some painters and different to other painters. 

Related problems include painter identification, painting authentication,
finding connections between painters etc. These problems have some similarities to the
problem described in this paper, but they are basicaly different. 

Classification in this paper will not consider semantics of a painting (i.e. the
objects and motifs in painting, data about year of production and painter
etc.) but only the painting technique. The following six genres are used:
realism, impressionism, cubism, fauvism, pointilism and na\"{\i}ve art. Example of
each genre is found on Figure \ref{fig:zanrovi}.

In addition to art critics who can distinguish different artistic genres,
there is a large number of people with limited knowledge of art who will
potentially incorrectly classify similar genres. The assumption of this paper
is that disagreements between people exist, considering classification in genres.

The next section gives an overview of related work. The third section describes
the extraction of previously mentioned elements of the painting.
Furthermore, the fourth section talks about tested methods of
classification and the fifth section describes the final results. Conclusion and
suggestions for future work on subject are given in the sixth section.


\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth]{zanrovi.png}
\caption{Example paintings of used genres (shown in order): Realism,
 impressionism, cubism, fauvism, pointilism, na\"{\i}ve art.}
\label{fig:zanrovi}
\end{center}
\end{figure}

\section{Related work}

There are few works that focus on the classification of paintings by
genre.

Article \cite{gunsel2005content} describes the classification of five genres:
classicism, impressionism, cubism, expressionism, surrealism.
The used features in this work are information about ``dark'' pixels in the
image (pixels whose RGB value lies in the interval [0,64]), information about
the edges in the image (coefficient of gradient, i.e.\ Sobel's operator),
luminance histograms (number of maxima in the histogram and the color of the
histogram peak) and features that allow elimination of the influence of light
conditions and image resolution, based on the gradient coefficients and
luminance mean value. Selection of features is based on the idea that certain 
genres include much more dark pixels than other genres and reach different
maxima in the luminance histogram.

Article \cite{zujovic2009classifying} used a similar approach as in
\cite{gunsel2005content}. Features that were extracted on gray scale
images are number of edges obtained by Canny edge detector and Gabor filters 
(the image is filtered several times with different parameters, and the mean 
value and variance of the sum of pixels is taken).
Other features in this work are dealing with color in the image through histogram
values for hue-saturation-value (HSV) color space.
Used images have different resolutions and quality. They are obtained from the
free art painting collection Artlex\footnote{\url{http://www.artlex.com}} and
by using Google search engine.
Classification of the following five genres is described: realism,
impressionism, cubism, abstract expressionism, pop art. Large number of
classifiers is tested in both works.

Article \cite{shamir2010impressionism} uses a slightly different approach.
Much more features are defined than in the previous two papers (various
edge statistics, object statistics, color histograms, Radon transformation
etc.). Classification via Fisher discriminant analysis is then conducted, using
the following three movements: impressionism, expressionism, surrealism. The
paper achieves better results than other works, but this is partly expected due
to smaller number of classes and possibly due to greater separability 
between them.

Our paper will use similar guidelines for feature extraction and some of the 
features described in these articles with moderate changes. 
Image database is obtained in the same way
(images found by Google search engine, with variety in quality and size).
Main difference from other papers is that they classify mostly 
the same genres that are usually very well mutually separable.
 
This work will, with the exception of already tested genres (impressionism, realism,
cubism), introduce several new genres that haven't been yet tested as
far as we know (fauvism, pointilism, na\"{\i}ve art). Some of the introduced
genres partially overlap, which further complicates the correct classification. 
For some of these genres large databases could not be obtained.

\section{Feature extraction}

This paper is combining features recommended in related work as well as 
proposing some new features. Baseline idea is to find features that improve genre
separability without considering the semantics of a painting. 
Features are based on color distribution or texture analysis since this is
how humans perceive and classify art (before taking semantics into
consideration). Genres were chosen in a manner that there are different and easily discerned
genres and several similar genres that are hard to separate.


\subsection{ Image filtering }

Image filtering is useful since it can reveal additional information hidden in
poor image quality or large number of details in image. Most common filters
convolve image elements with filter kernel. The following filters have been used: blur, sharpen
and edge detection. Some features require several filters applied in a sequence.
An example of a sharpen filter is given in Figure~\ref{fig:ostro}.

\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth]{sharp.png}
\caption{On the left is the original image, and on the right is the image after sharpening.}
\label{fig:ostro}
\end{center}
\end{figure}

\subsection{ Color based features }

Certain genres have mostly dark or mostly light paintings, some genres have a
wide color palette while others use a specific color tone. Colors can be
considered through several color models and this paper uses two models or color spaces:
red-green-blue (RGB) and hue-saturation-value (HSV). Luminance component of
color is calculated separately. Basic tool for color space analysis are
histograms (an example histogram generated for the original image in Figure~\ref{fig:ostro}
can be seen in Figure~\ref{fig:histogrami}). Histograms show color component
distribution of an image.

Basic histogram feature is mean value for intervals of the normalized histogram.
For a given interval $k$, mean $\mu^{color}_k$ is calculated as:
\begin{equation}
\mu^{color}_k = \frac{\sum_{i = k \cdot l}^{(k + 1)\cdot l} H(i)}{l \cdot H_{max}}
\label{eqn:mu}
\end{equation}
where $l$ is interval length, $H(i)$ is the normalized histogram value for value $i$ of a 
color component. $H_{max}$ is the maximum value in the normalized histogram.
E.g. green color histogram in Figure~\ref{fig:histogrami} has the following mean
values: $0.38$, $0.83$, $0.72$, $0.55$, $0.41$, $0.31$, $0.11$ and $0.02$ for $l=32$. 
These eight values show that the first half of the histogram dominates and that tells us
that green colors of the image in Figure~\ref{fig:ostro} are mostly dark.

In similar manner as in \cite{gunsel2005content}, number of local maxima is considered on
any given histogram ($LM^{color}$).
E.g. luminance histogram in Figure~\ref{fig:histogrami} has $4$ local maxima.
Depending on the position of every maximum, a conclusion can be made on general image
lightness. From this luminance histogram example, we can say that the image from
Figure~\ref{fig:ostro} is mostly dark or mildly light.

In addition to the number of local maxima we are also looking for the position
of the highest value (i.e. peak). For the given example, luminance histogram has
the value of $\varphi^{luminance}=0.17$ for the highest local maximum. Value
range is $[0,1]$ with darker pictures closer to $0$ and brighter pictures closer to $1$.

It is also possible to count the number of pixel values in a given
interval divided by the number of pixels in the image to get various pixel
ratios. One possible use of this feature is counting pixels in
luminance range of $[0,64]$ (dark pixels) as in \cite{gunsel2005content}. The result is a
ratio of dark pixels in an image, i.e. ``darkness'' of the image.

Table \ref{tab:tablicaBoje} contains some of the values for one picture from
each genre. Although generalization based on a single sample is inaccurate, it
serves as an example of difference between genres based on color features and
that those features could be important for classification. More on that in
section \ref{sec:rezultati}.  

Main differences between genres are based on luminance. Fauvism, cubism,
pointilism and na\"{\i}ve art use more transitions in intensity, while impressionism
and realism use fine and even transitions. That results in reduced number of
local maxima in luminance histograms for those two genres. Some genres have
certain dominating colors. Na\"{\i}ve art mostly depicts natural motifs so green
color is prevalent, while realist artists have darker pictures that lean toward
red tones. Cubists like intensive colors and have high maxima for each color
component and saturation values are high. In addition to numerical values in
Table \ref{tab:tablicaBoje} those characteristics are easy to notice in
Figure \ref{fig:zanrovi}.

\begin{table*}
\caption{Some values of color features.}
\label{tab:tablicaBoje}
\begin{center}
\begin{tabular}{lccccccc}
\toprule
Genre & $Max(\mu^{red})$ & $Max(\mu^{green})$ & $Max(\mu^{blue})$ & $LM^{lum}$ & $\varphi^{lum}$ & $\varphi^{hue}$ & $\varphi^{sat}$ \\
\midrule
Fauvism & $0.61$ & $0.68$ & $0.72$ & $4$ & $0.08$ & $0.04$ & $0.61$\\
Cubism & $0.81$ & $0.83$ & $0.97$ & $4$ & $0.17$ & $0.08$ & $0.65$\\
Impressionism & $0.94$ & $0.76$ & $0.88$ & $2$ & $0.66$ & $0.57$ & $0.70$\\
Realism & $0.73$ & $0.46$ & $0.68$ & $2$ & $0.90$ & $0.14$ & $0.92$\\
Pointilism & $0.03$ & $0.82$ & $0.73$ & $4$ & $1.00$ & $0.08$ & $1.00$\\
Na\"{\i}ve & $0.47$ & $0.54$ & $0.69$ & $6$ & $0.07$ & $0.46$ & $0.07$\\
\bottomrule
\end{tabular}
\end{center}
\end{table*}

\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth]{histogrami.png}
\caption{Example of red, green and blue color and luminance histograms.
On abscissæ are values of red, green, blue and luminance intensity
in interval of $[0,255]$, while ordinates have normalized histogram values in range of $[0,1]$.}
\label{fig:histogrami}
\end{center}
\end{figure}

\subsection{Features based on texture}

Texture of a painting is characterized by brush strokes, shapes and general
details. Sharpened image in Figure \ref{fig:ostro} shows that the artist uses very
rough textures with more brush strokes than originally perceived. Some genres
use rough painting style with visible edges, while others have smoother
transitions. Excellent tools for extracting features based on texture are
filters for edge detection and detail enhancement.

Main feature of a texture is the amount of edges in an image. Canny filtered
image is black where there are no edges (example in Figure
\ref{fig:canny}). By counting colored pixels and dividing by total number of
pixels we get the ratio of edges in an image. For the given example using Canny
edge filter we get the ratio of $0.09$. Canny filter thresholds are the same as
in \cite{gonzalez18075digital}, as implemented in the \emph{Java Image
Processing Toolkit} library\footnote{http://sourceforge.net/projects/jipt/}.

We extended edge detection with additional image preprocessing, such as
sharpening or blurring. Sharpening distinguishes smaller -- more detailed edges,
while blurring removes them and leaves only more substantial edges.
Using preprocessing on the given example we get $0.11$ edge ratio for
the sharpened image and $0.04$ for the blurred image.

Blur and sharpen filters are also used to determine how sharp or blurry an
actual image is. By using the filters, subtracting the original image and
counting colored pixels it is possible to determine the sharpness of an image.
Even though this feature is related to the edges, it looks at their surroundings
rather than the edges themselves.

It is possible to combine filters, change their order and perform operations
before and after filtering. Example of one such combination is filtering an
image with two different sets of filters and calculating the ratio of
differences between them. Interpretations of such combinations can be
complicated, but they prove to be very useful in classification.

Some images are horizontally or vertically symmetric. Measure of symmetry is
determined by subtracting images mirrored by a given axis. Counting dark elements
on a filtered image gives a ratio of elements that are mirrored on the other
side of an axis.

\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth]{canny.png}
\caption{Canny filter example}
\label{fig:canny}
\end{center}
\end{figure}

\section{Classification}

Several classifiers with different settings have been tested using Weka
tool.\footnote{\url{http://www.cs.waikato.ac.nz/ml/weka/}} We used the following
classifiers, chosen because of their speed and accuracy: artificial neural network (ANN),
 random forest \cite{breiman2001random}, sequential minimal optimization for
 support vector machine (SMO) \cite{platt1999sequential}, k nearest neighbors (k-NN)
 and decision table.

\section{Results}
\label{sec:rezultati}

\begin{table}
\caption{Image database.}
\label{tab:baza}
\begin{center}
\begin{tabular}{lc}
\toprule
Genre & Number of images \\
\midrule
Realism & $166$\\
Impressionism & $165$\\
Cubism & $95$\\
Fauvism & $140$\\
Pointilism & $67$\\
Na\"{\i}ve art & $60$\\
\bottomrule
Total & $693$\\
\end{tabular}
\end{center}
\end{table}

Examples used for training and testing were obtained using Google search
engine and Artlex image database. As genre labeling reference we were using
Artlex database and Wikipedia.
Number of images per genre (class) and overall number of images is shown
in Table \ref{tab:baza}.
Due to time and memory requirements, large images were scaled so their
larger side is equal to 1024 pixels (factor was 1024 by larger side).

It is quite possible that there is a certain number of images whose genre
is not correctly determined at our sources (Artlex, Wikipedia), but this is
expected and acceptable as determination of the genre of image is not unique.
In addition,
some of the genres partially overlap (e.g. impressionism vs. fauvism or 
pointilism as branches of neo-impressionism). Unfortunately, this problem
will affect the test results. Selected examples vary in size and quality
(most of images are lower quality), which contributes to the use of these
methods in everyday life, such as photographing paintings using mobile phones.

For validation we have used 10-\emph{fold} cross-validation, and \emph{a priori}
probability (probability that image from database belongs to class $c$ which is
equal to $N_c / N$, where $N_c$ is number of images in class $c$ and $N$ is number
of all images in database) as the reference method. Reason for that choice of
reference method is the assumption that most people are not very good at
recognizing the genre of a painting, so they will guess when genres overlap.

The results of the reference method range from $8.66\%$ accuracy for na\"{\i}ve
art as a genre with the lowest number of examples to $23.95\%$ accuracy for
realism, which has the largest number of examples.

The complete system contains 68 features:
\begin{enumerate}
\item mean values of histogram intervals for RGB and HSV color and luminance
(50 features),
\item number of local maxima for the HSV color and luminance histograms (4
features),
\item positions of the peak for HSV color and luminance histogram (4
features),
\item ratio of dark pixels,
\item amount of edges extracted using Canny edge detector, with and without previous blurring/sharpening (4 features),
\item estimation of image sharpness (2 features),
\item vertical and horizontal symmetry (2 features),
\item ratio of sharpened edges and real edges.
\end{enumerate}

Overview of the classification performance using
different classifiers and different combinations of features is shown in
Table \ref{tab:rez}.
Classification performance is expressed using $F_1$ measure which is harmonic mean of precision ($P$) and recall ($R$):
\begin{equation}
F_1 = 2\cdot \frac{P\cdot R}{P + R},
\end{equation}
while precision is defined as portion of the correctly classified images, and recall is the portion of the correctly assigned classes:
\begin{align}
P &= \frac{\rm{TP}}{\rm{TP} + \rm{FP}},\\
R &= \frac{\rm{TP}}{\rm{TP}+\rm{FN}},
\end{align}
where $\rm{TP}$ is number of images of class $c$ that were classified as class $c$, $\rm{FP}$ is number of images which are not part of class $c$ but they were classified as class $c$ and $\rm{FN}$ is number of images of class $c$ that were not classified as class $c$.
Features connected with color are shown at the column labeled ``Color,''
they were previously listed as 1st, 2nd, 3rd and 4th feature.
Features associated with texture properties are shown in column
``Texture,'' and are listed as 5th, 6th, 7th and 8th feature. Column ``All''
includes all listed features.

From result analysis it is clear that the features derived from texture,
although their small number, contribute more to successful classification
than features derived from color. The reason is the fact that
genres do not differ greatly by color properties but more by ways of
painting shapes and image semantic. Some paintings under the same genre
may be predominantly green and bright, and some dark and red. On the other
hand, strong edges are typical for cubism while lighter edges and mixing
colors are more typical for impressionism.
Use of the color based features could be more successful for other genres,
like those listed on related works.

It is interesting to note that the performance of classifiers based on
rules (decision tables), trees (random trees) and distance (k-NN) decreases
when using all of the features. Further experiments have shown that the
reduction of the number of features (e.g., removing color histograms) increases
performance of those classifiers. This indicates that there is a problem 
with too large dimensionality of feature space. Although these
classifiers have better performance on reduced feature set, they still do not
surpass the ANN and the SMO classifiers.

Relationship between genres is shown in Table \ref{tab:zabuna}.
Pointilism is a genre with the best separability. It is followed by cubism,
na\"{\i}ve art and realism as satisfactory separable genres. Fauvism and
impressionism have the worst statistics, which could be explained from the fact that those
genres originate from approximately the same period and many artists who have
represented one movement were connected with other movements. Fauvism is often 
mixed with cubism, and the reason for this is relative closeness of genres, 
which is visible at Figure \ref{fig:zanrovi} where both paintings are full 
of strong lines and large areas of various intensive colors. A lot of genres 
are classified as realism, which probably arises from the monotony of color 
and shape of that genre.

After removing impressionism from the system, SMO classifier achieved a success
rate of $68\%$, while removing fauvism SMO achieves success rate of $64\%$.
That indicates that impressionism may be too similar to other genres, therefore
we need a feature or set of features that would improve separation of that genre from
the others. Regarding the low performance of features based on color, it
is more likely that such features could be found in the analysis of textures,
edges and shapes.

In \cite{zujovic2009classifying} maximum achieved accuracy is $68.3\%$.
A maximum accuracy in \cite{shamir2010impressionism} is $91\% $, but their
method is tested on only 3 genres. In \cite{gunsel2005content} maximum
accuracy has reached over $90\%$, but \cite{zujovic2009classifying} cites the
same article and note that their maximum accuracy is only $49.8\% $.
Therefore we can assume that there is a problem at comparing methods on
different genres. The features can be highly dependent on the genre, so
comparing different methods is problematic. With the achieved $F_1$ measure
at amounts of $60.2\%$, $64\%$ and $68\%$, we find our work commensurate with
related works.

\begin{table}
\caption{Classification results for all chosen features, $F_1$ measure in
percents.}
\label{tab:rez}
\begin{center}
\begin{tabular}{lcccc}
\toprule
Classifier 	& Color 		& Texture 	& All 		\\
\midrule
ANN 			& $37.7$ 	& $55.0$	& $56.6$	\\
RandomForest 	& $38.2$ 	& $53.2$	& $50.7$	\\
SMO 			& $42.5$ 	& $52.1$	& $60.2$	\\
k-NN ($30$) 	& $38.8$ 	& $52.9$	& $46.4$ 	\\
DecisionTable 	& $36.6$ 	& $47.0$	& $44.3$ 	\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\begin{table}
\caption{Confusion matrix for SMO classifier and all features.}
\label{tab:zabuna}
\begin{center}
\begin{tabular}{lcccccc}
\toprule
			& C		& F		& I		& N		& P		& R 	\\
\midrule
Cubism		& $53$	& $9$	& $7$	& $4$	& $0$	& $17$		\\
Fauvism		& $14$	& $50$	& $13$	& $7$	& $1$	& $5$		\\
Impressionism& $7$	& $14$	& $37$	& $14$	& $2$	& $16$		\\
Na\"{\i}ve art	& $5$	& $5$	& $6$	& $35$	& $4$	& $5$		\\
Pointilism	& $0$	& $2$	& $4$	& $1$	& $59$	& $1$		\\
Realism 	& $7$	& $3$	& $12$	& $9$	& $0$	& $59$		\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\section{Conclusion}
This paper presents an effective approach to classification of paintings by
art genre. The features are based on the color and texture of an image.
Due to large number of genres, partly overlapping genres and low
quality of examples, we can determine that it is possible to achieve
better results by using better database or choosing other genres. In future 
work, it would be useful to develop new features describing the image texture.

We can conclude that computer's ability to analyze paintings raises
important questions about the perception of applied arts. We can
ask ourselves could computer systems become art critics and could they do more
than just evaluate art -- could they create it?


\section*{Acknowledgment}
We would like to thank Ivan Krišto for helping us in improving this paper.

\bibliographystyle{IEEEtranS}
\bibliography{IEEEabrv,su2010}
\end{document}
